When vast amounts of data are present and are continuously entering in a data store (which might even be distributed), accurate and quick analysis of the data in the data store is of crucial importance for users who want to act upon information contained therein.
It is thereby important that the response time on a user request is within a reasonable (e.g. a few seconds) time limit. The response must moreover be based on an accurate representation of the data which is present in the data store. The data store may contain a large variety of data, it may be distributed and the stored data may be unstructured. User requests may therefore require a thorough analysis of the data store which has a direct impact on the response time.
Data processing systems are therefore facing the problem of, on one hand, real-time response time requirements, and, on the other hand, thorough analysis requirements on distributed data stores containing a vast amount of data which is continuously increasing.
US2013/0013552 A1 discloses an interest-driven Business Intelligence system. In this data processing system raw data is stored in a raw data storage, metadata is stored in a metadata storage, and the system comprises an interest-driven data pipeline that is automatically compiled to generate reporting data using the raw data. The interest-driven data pipeline is compiled based upon reporting data requirements automatically derived from at least one report specification defined using the metadata. US2013/0013552 A1 thereby wants to have an improved response time, and thus improved interactivity, compared to existing systems used to store large volumes of data. Instead of using data pipelines which need to be changed by engineers each time the data requirements are changing, an interest-driven data pipeline is automatically compiled when the reporting data requirements are changed.
There is, however, still room for more efficient data processing systems for big data analytics (for example with regard to response time, accurateness, and/or flexibility).